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The Race to the Top—Early Learning Challenge (RTT-ELC) grants program, sponsored by 
the U.S. Department of Education and U.S. Department of Health and Human Services, aimed to 
improve children’s access to high quality early learning and development programs. RTT-ELC 
awarded $520 million in the first of three rounds of grants to help states develop and implement 
systems that rate early learning and development programs on quality and help them improve. 
These systems are known as tiered quality rating and improvement systems (TORTS). TORIS rate 
programs based on state-defined quality standards. Programs at the lowest level meet a basic level 
of quality, whereas programs at the top levels meet the highest standards of quality as measured 
by one of these rating systems. 


A key objective for RTT-ELC was to increase the number of early learning and development 
programs in the top rating levels of the TORIS, and to increase the number of children with high 
needs, such as those from low-income households, enrolled in these programs. To meet this 
objective, states could try to encourage more programs to participate in TORIS and seek high 
ratings. 


This brief examines states’ progress on this objective by describing patterns of TORIS 
participation and ratings for the nine Round I states during the grant period (2012 to 2016). The 
brief presents patterns by state and for four types of programs: (1) state-funded prekindergarten 
(pre-K) programs, (2) Head Start programs, (3) licensed centers that received child care subsidies, 
and (4) licensed centers that did not receive subsidies. The first three types of programs—state- 
funded pre-K, Head Start, and licensed centers that received subsidies—serve low-income 
children; the last type might be less likely to serve low-income children because they do not receive 
public funding to do so. Key findings, based on administrative TORIS data from the nine states, 
include: 


e TORIS participation in all states and among all types of programs increased from 2012 to 
2016. Programs that served low-income children—state-funded pre-K, Head Start, and 
licensed centers that received subsidies—participated at much higher levels than licensed 
centers that did not receive subsidies, which might not serve low-income children. 


e Patterns in the percentage of programs in the top two rating levels differed substantially 
by state from 2012 to 2016. This finding could be due, in part, to various differences across 
states, including differences in states’ TORIS characteristics and policies and the 
composition of programs that participated in each state. 


e Most states increased the percentage of programs at the top rating levels. In particular, 
most states expanded TORIS participation among two types of programs (state-funded pre- 
K and Head Start programs) that serve low-income children. These programs tended to 
enter TORIS at high levels either by meeting TORIS standards or through policies 
(alternative pathways and automatic ratings) that grant them credit for certain TORIS 
standards because they meet them through external quality standards. 
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e The availability of high-rated programs for low-income children increased over time in the 
nine RTT-ELC states. Notably, licensed centers that received subsidies moved from lower 
to higher rating levels over time. However, most children served by licensed centers that 
received subsidies were not enrolled in programs at the top levels by 2016. 


Providing young children from birth to age 5 with high quality early learning and development 
experiences, particularly in formal center-based and preschool settings, yields significant 
benefits—especially for children from low-income and disadvantaged households (Dearing et al. 
2009; Weiland and Yoshikawa 2013). These benefits can include short-term improvements in 
social skills, behavior, and language skills, as well as long-term effects, such as pursuing more 
years of education and earning higher income (Campbell et al. 2002; Heckman 2011; Duncan and 
Magnuson 2013). To improve the quality of early learning and development that children 
experience, states created TQRIS that establish quality standards, encourage programs to 
participate, and rate programs on the standards (Zellman et al. 2011). These systems also provide 
incentives for programs to earn higher ratings and publicize the ratings of individual programs to 
help parents choose better programs for their children. 


Developing and implementing TQRIS was a key aspect of the RTT-ELC program. Through 
TQRIS, RTT-ELC sought to strengthen the quality of 
early learning and development programs by | Box1.RTT-ELC’s five TQRIS objectives 


promoting progress on five objectives (Box 1), . Developing and adopting a common, 
including increasing access to high quality programs, statewide TQRIS 
particularly for children from low-income and . Promoting participation in the TORIS 
disadvantaged households (Applications for New . Rating and monitoring early learning 
Awards; Race to the Top-Early Learning Challenge and development programs 
2011). The Departments of Education and Health and . Promoting access to high quality 
Human Services awarded RTT-ELC grants through DiS e et een ere WAU ie uiles de 
three rounds of competition to help states implement by: 
new TQRIS or increase participation and ratings in i saan nae ae 
established TQRIS. Round 1 grants, which are the Fone ends Be ere eae 
focus of this brief, were awarded in December 2011 to ; 

j : ‘ ncreasing the number and 
nine  states—California, Delaware, Maryland, percentage of children with high 
Massachusetts, Minnesota, North Carolina, Ohio, needs who are enrolled in 
Rhode Island, and Washington. Of the Round | states, programs that are in the top levels 
five had statewide TQRIS by 2011, the year in which | 5. Validating the effectiveness of the 


RTT-ELC grants were awarded. By 2015, eight of the MGR 
states had statewide TQRIS, and the ninth state— 
California—had locally administered TQRIS but statewide adoption was underway. 


The Institute of Education Sciences at the Department of Education initiated a study to learn 
about TQRIS in the nine states that recetved RTT-ELC Round | grants and to inform ongoing 
TQRIS development. The study focused on center-based early learning and development programs 
that served preschool-age children (these programs might have also served infants, toddlers, and 
school-age children). The study generated a series of reports and briefs. The first report examined 
progress on the first three TQRIS objectives (outlined in Box 1) by describing the development, 
structure, and characteristics of TQRIS in the Round | states. It found that these states made 
progress in developing and implementing statewide TQRIS (Kirby et al. 2017). However, they 
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varied substantially in the ways they promoted participation in TQRIS, defined quality standards, 
verified that programs met the standards, and calculated ratings. 


This brief contributes to the larger study by describing states’ progress on the fourth 
objective—promoting access to high quality programs for children with high needs. It focuses on 
the parts of the objective that could be examined with the data available: increasing the number of 
programs at top levels of the TQRIS and increasing the number of children (including those from 
low-income households) served by top-rated licensed centers that received subsidies. Data were 
not available on the total number of children or number of children from low-income households 
that state-funded pre-K or Head Start programs served. Data were also not available for the full 
range of children with high needs (such as those who have disabilities or developmental delays, 
are English learners, or are migrant, homeless, or in foster care). Thus, although the brief provides 
some information about the availability of top-rated programs for low-income children, it does not 
provide complete information about increased access to high quality programs for children with 
high needs. 


To achieve an increase in the number of programs at top levels, the nine Round 1 RTT-ELC 
states could try to encourage more programs to participate in TQRIS and seek high ratings. This 
brief describes the patterns in TQRIS participation and ratings among programs in the nine Round 
1 states during the RTT-ELC grant period from 2012 to 2016. It examines programs’ attainment 
of the highest TQRIS rating levels and how programs reached those levels—at the time of entry 
or by progressing to higher ratings after entry. It also focuses on patterns of participation and 
ratings for types of programs that serve children from low-income households. For licensed centers 
that received child care subsidies—the only type of program serving low-income children for 
which states consistently provided data on the number of children served—the brief presents the 
percentages of children served in centers at the top rating levels. 


Future work that is planned for this study will further examine RTT-ELC states’ progress on 
the fourth and fifth TQRIS objectives (Box 1). This work plans to provide additional information 
about (1) patterns of TQRIS ratings across states with different TQRIS characteristics and policies 
and (2) findings from states’ TQRIS validation studies. 


Research Questions 


To understand potential changes in children’s access to high quality programs in the nine 
Round | states under RTT-ELC, we examined four research questions: 


1. How did TQRIS participation change during the RTT-ELC grant period from 2012 to 
2016 in the nine states? What was the composition of programs that participated in TQRIS 
by 2016? 


2. How did the percentage of programs in the top TQRIS rating levels change from 2012 to 
2016 in the nine states? Did the distribution of ratings shift toward the higher levels over 
time? 


3. What were the patterns in rating levels at TQRIS entry and improvement in rating levels 
among different types of programs? How did TQRIS participating programs in the nine 
states achieve the top two rating levels by 2016? 


4. Did states make progress in promoting availability of high quality programs for low- 
income children? 
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For all of these questions, we also examined how TQRIS participation, ratings, and movement 
varied across program types and states. Different types of programs serve children of different 
ages and from different income groups, and they face different incentives to participate in TQRIS 
and earn higher ratings. States also differ in how they structure TQRIS and in the policies they use 
to try to encourage programs to participate and improve. 


The answers to these research questions help shed light on states’ progress toward increasing 
children’s access to high quality programs. First, to increase the number of programs at top levels 
of TQRIS, states must encourage programs to participate. Second, the number of programs that 
receive high TQRIS ratings provides information about the prevalence of programs that met 
quality standards (either state-defined TQRIS standards or other external quality standards that 
states may use as a proxy for meeting certain TQRIS standards or rating levels). Third, programs’ 
ratings at entry and the movement of programs from low to high rating levels provide information 
about how programs achieve top rating levels; improving programs’ ratings is a key way that states 
aim to increase the availability of high quality programs. Finally, for children from low-income 
households to have access to high quality programs, those programs must first be available. 


In Boxes 2 and 3, we describe the data sources, methods, and samples that we used to answer 
these questions. 


Box 2. Data sources 


Telephone interviews with TQRIS administrators in the nine states provided information about TQRIS structure, policies, 
and practices from 2012 through 2016. We conducted two rounds of interviews. The first round, from October 2014 
through April 2015, collected information on TQRIS characteristics from the start of the RTT-ELC grants in 2012 through 
the time of the interview. The second round, in December 2016 and January 2017, confirmed or updated this 
information. 


Administrative data from the nine states provided information about TQRIS participation and ratings from the start of 
the RTT-ELC grants in 2012 through June 2016. Data for 2016 are for only half the year (from January through June). 
Data from California are from the 16 counties in which the state implemented its TQRIS beginning in 2012. California 
provided data for these counties for 2014 to 2016. All other states have statewide TQRIS and we collected data for each 
of those states as a whole. In Ohio, data from before and after October 2013 (when the state moved from a system with 
three rating levels to one with five levels) were not comparable. For these reasons, we only used data from Ohio and 
California from 2014 to 2016. 


We combined the TQRIS data with other administrative data, including licensing and child care subsidy data, and state- 
funded pre-K and Head Start program lists and data, to build a longitudinal picture of early learning and development 
programs over the study period, including those that did and did not participate in TQRIS. All states provided data on 
state-funded pre-K, Head Start, and licensed programs that did and did not participate in TQRIS. However, most states 
did not provide data on license-exempt programs that were not either state-funded pre-K or Head Start programs (which 
are often considered license-exempt when operated in public schools), particularly if they did not participate in TQRIS. 
Only three states allowed license-exempt programs that were not receiving state pre-K or Head Start funds to participate 
in TQRIS. Such programs could include those administered on Tribal lands or on military bases, or those affiliated with a 
religious program. These programs accounted for less than 1 percent of TQRIS participating programs in 2016 in two of 
the three states and less than 5 percent in the third state. 


Data on the number of children enrolled (enrollment), the number of children a program served (capacity), and the 
number of children with high needs were limited. States provided complete data on capacity only for licensed programs. 
Most states did not provide complete data on enrollment or the number of low-income children supported through child 
care subsidies in a center-based program. No states could provide center-level data on the number of children supported 
with either Head Start or state pre-K funds. States also could not provide data on the full range of children with high 
needs (such as those with disabilities). 


Accreditation data from the National Association for the Education of Young Children (NAEYC) provided information 
about the accreditation status of early learning and development programs in each of the nine states as of June of each 
year from 2012 to 2016. 


For more information about the data sources, see Appendix A. 
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Box 3. Methods and samples 


We used descriptive analyses to describe programs’ participation and ratings, overall and by type of program and state. 
For these analyses, we used the full population of programs, as opposed to a sample. Thus, any observed differences 
reflect actual differences, not estimates based on statistical tests. 


Due to data limitations, all analyses exclude license-exempt programs that were not either state-funded pre-K or Head 
Start programs (which are often considered license-exempt when operated in public schools). 


Participation 


To examine participation, we used data on all programs (from the four program types) in each state, including those that 
did not participate in TQRIS. We then calculated the percentage that participated in TQRIS. The number of programs in 
the analyses ranged from 14,287 in 2012 (in seven states, excluding California and Ohio) to 28,355 in 2016 (in nine 
states) (Appendix A, Table A.1). 


Analyses that combined data across states and compared 2012 with 2016 excluded California and Ohio because they 
did not have usable data for the entire period. This restriction ensured that comparisons between 2012 and 2016 reflect 
true changes in participation across years and not changes caused by differences in data availability. The number of 
programs in the analyses for the seven states ranged from 14,287 in 2012 to 14,100 in 2016. 


Ratings 


Analyses of ratings included only programs that participated in TQRIS from among the four program types and had ratings 
in each year. The number of programs with ratings in each year ranged from 5,414 in 2012 (in seven states) to 13,357 
in 2016 (in nine states). 


We used five standardized rating categories that describe the position of rating levels within each state’s rating structure. 
Seven of the nine states used a five-level rating structure and two states used four levels. Rating levels were not directly 
comparable across states, even among states with the same number of ratings, because states included different quality 
components and calculated the ratings differently (Kirby et al. 2015). The four-level rating structure only uses four of the 
categories: in this structure, 1 is the lowest level, 4 is the highest level, and 2 and 3 are the second-lowest and second- 
highest levels, respectively. There is no middle level in the four-level rating structure, because the exact middle of 1 and 
Ais 2.5. 


The standardized categories align the top two rating levels across the four- and five-level structures, following the most 
common and conservative definition of “top tiers” that RTT-ELC states used. In their grant reports, the majority of RTT-ELC 
states only included ratings in the top two levels (4 and 5 in the five-level structure and 3 and 4 in the four-level structure) 
in their “top tiers.” These categories standardized the ratings as shown in the table below. 


Standardized rating level 


category Rating in 4-level rating system Rating in 5-level rating system 
Highest level 4 5 
Second-highest level 3 4 
Middle level na. 3 
Second-lowest level 2 2 
Lowest level 1 al 


n.a. = not applicable. 


The analysis examined the percentage of programs that received each rating level category and the percentages in the 
two highest levels combined. 


To understand how programs reach the two highest levels, we examined ratings at entry and movement to higher ratings. 
For programs that reached the highest levels before the analysis period began (2012 for seven states and 2014 for 
California and Ohio), we did not observe how they reached these levels (at entry or through movement). The entry analysis 
included programs that entered during the analysis period (9,658 programs). The movement analysis used program 
ratings from 2012 to 2016 and included 14,042 programs. It (1) included programs that participated in TQRIS for any 
period of time (longer than one year) within the analysis period (even if they exited before 2016), and (2) excluded 
programs that did not have enough time to change ratings (that is, those that entered TQRIS in 2016. 
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Types of Participating Programs and TQRIS Policies 


The types of programs that participate in TQRIS and the policies that are intended to promote 
participation and encourage quality improvement could contribute to patterns in participation and 
ratings. Understanding the different program types and TQRIS characteristics and policies 
provides important context for interpreting the patterns in participation, ratings, and improvement 
that we present. It also underscores how these patterns might translate into increased availability 
of high quality programs for low-income children. 


Program types. The landscape of center-based early learning and development programs that 
serve children ages 3 to 5 is quite diverse. Programs differ in the ages of children they serve and 
the degree to which they serve low-income or high-needs children (Table 1). State-funded pre-K 
and Head Start programs specifically target preschool-age children and fully, or predominantly, 
serve low-income children, respectively. Licensed centers can serve children from birth through 
age 12; in this study, we included only those that served preschool-age children. Licensed centers 
that received child care subsidies served at least some low-income children; we could not 
determine from the data whether licensed centers that did not receive subsidies served low-income 
children or the numbers of low-income children that programs served. Other differences among 
programs include the type of setting (such as within a public school or a community-based 
organization), the type and source of their licensing or regulation, and the types of funding or 
revenue they receive. 


Many programs receive funding from a mix of sources, so categorizing each program as a 
single type might not fully capture its nature. For example, a community-based center could have 
Head Start classrooms for children ages 3 to 5 and state-funded pre-K classrooms for 4-year-olds. 
It could also receive Child Care and Development Fund subsidies to support low-income children 
from birth to age 12 and receive full or partial tuition directly from parents. However, to learn 
about patterns across programs and how TQRIS policies might influence them, we created 
mutually exclusive program types that include state-funded pre-K programs, Head Start programs, 
licensed centers that received child care subsidy funding, and licensed centers that did not receive 
any type of public funding examined in this analysis (see Appendix A, Table A.2 for more 
information about how we created these program types and their licensing status and funding 
sources). 


Not all Head Start and state-funded pre-K programs must hold a license from the state child 
care licensing entity. However, these programs must follow standards set by the Federal Office of 
Head Start or state department of education, respectively. These standards include components of 
quality that are similar to those of TQRIS standards and often set requirements that are comparable 
to those for the highest TQRIS rating levels. For example, state-funded pre-K programs in the nine 
states require the use of early learning development standards and approved curricula (Barnett et 
al. 2017). 


Programs that are not administered by public schools can receive accreditation from the 
National Association of the Education of Young Children (NAEYC). NAEYC accreditation means 
that a program has met standards for quality in 10 areas including curriculum, assessment of child 
progress, physical environment, and leadership and management, among others (NAEYC 2017). 
We did not classify NAEYC accredited programs as a specific program type for these analyses. 
Instead, we included this accreditation as a program characteristic because some TQRIS policies 
specifically target NAEYC-accredited programs (Table 1). 
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Table 1. Characteristics of the types of early learning and development programs in the nine Round 1 


RTT ELC states 


Type of 
program 


State- 


Children served 
All states serve 4-year-olds; 


Setting 
Community-based 


Licensing or regulation 
Regulated by state 


avo) itemivlarel fared 
received 


State pre-K 


Percentage of 
programs that 


had NAEYC 
accreditation in 
2016 


2% (only applies 


funded 5 states also serve 3-year- or public school departments of education funds. to programs 
pre-K olds. settings using state program May also have delivered in 
Targeted to low-income standards. received Federal _ licensed 
children in 7 states. May be licensed in Head Start centers) 
community-based settings funds and child 
or considered license- care subsidies. 
exempt when delivered in 
public school settings. 
Head Start Targeted to low-income Community-based Regulated by the Federal Federal Head 5% (only applies 
children ages 3 to 5. or public school Office of Head Start using Start funds. to programs 
settings Head Start Program May also have delivered in 
Performance Standards. received state- licensed 
May be licensed in pre-K funds and centers) 
community-based settings child care 
or considered license- subsidies. 
exempt when delivered in 
public school settings. 
Licensed Range from birth to school- Community-based Licensed by child care Federal Child 8% 
centers, age (up to age 12).@ settings licensing entity in the state. Care and 
received Subsidies support low- Development 
subsidies income children; may also Fund. 
serve children from higher- State-funded 
income levels. child care 
subsidies. 
Licensed Range from birth to school- Community-based Licensed by child care None.® 4% 
centers,no age (up to age 12).@ settings licensing entity in the state. 
subsidies 


May serve low-income 
children but data available 
on public funding sources 
do not indicate this. 


Source: Document reviews and telephone interviews conducted from December 2016 to January 2017, and administrative data collected 
from NAEYC and the Round 1 RTT-ELC states from October 2016 to April 2017. 


aThe analyses included only licensed centers that served preschool-age children. 


>Centers did not receive the public funding examined in this analysis including child care subsidies, Head Start, or state pre-K funds. 


TQRIS characteristics and policies. TQRIS first emerged in the late 1990s, expanded 
somewhat in the early 2000s, and flourished under RTT-ELC. By the end of 2017, 44 states had 
TQRIS (BUILD Initiative 2017).! TQRIS rate early learning and development programs against 
state-defined quality standards that include components such as licensing compliance, quality of 
the learning environment, and qualifications of the workforce. Based on meeting the standards, 
programs receive an overall rating level. In the Round | RTT-ELC states, the overall rating ranged 
from | to 4 in two states and | to 5 in seven states. 


The meaning of each rating level—what it measures and how—is not the same across states. 
States define TQRIS standards and rating levels differently (for more information about how the 
nine Round 1 RTT-ELC states defined rating levels, see the first report from this study, Kirby et 
al. 2017). For example, the nine Round 1 RTT-ELC states use anywhere from 6 to 12 components 
for their highest rating level. Even when multiple states use a particular TQRIS component (such 
as the quality of the learning environment), each state might differ in the rating level for which 
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that component is required. The commonality across the states is that the higher the rating, the 
higher the level of quality of the program based on the state’s TQRIS standards. 


States also differ in the characteristics and policies that they define for their systems, which 
could influence programs’ participation and the ratings they receive. The nine Round 1 RTT-ELC 
states vary widely in how they chose to structure TQRIS. These nine states are not representative 
of all TQRIS, but they do reflect the wide variation in characteristics and policies that exists across 
the 44 current TQRIS (BUILD Initiative 2017).? 


The nine Round | RTT-ELC states vary across the following characteristics, which could 
influence programs’ participation and the frequency and ease with which programs can achieve 
higher ratings (Table 2): 


e Length of implementation. A state system needs about two to four years, on average, to 
achieve a steady state of implementation in which the policies, practices, and supports are 
in place and running smoothly (Metz and Bartley 2012). Mature systems might have more- 
established policies that aim to encourage and support movement through the ratings. At 
the beginning of the study period in 2012, four states had implemented TQRIS statewide 
for two years or more. North Carolina had the longest-running TQRIS, which it adopted in 
1999. Other states expanded or launched TQRIS closer to the start of the RTT-ELC grants. 
Two of the nine states expanded TQRIS from a pilot to statewide at the start of RTT-ELC 
in 2012 and two others first launched TQRIS concurrently with RTT-ELC. 


Table 2. TQRIS characteristics in RTT-ELC Round 1 states 


Length of TQRIS 
implementation in Rating 
2012 structure? Entry requirement at lowest rating level Validity period for ratings 
California First year Points None 2 years 
Delaware 4 years Hybrid Licensed centers that received subsidies 1-3 years by rating level 
Licensed centers with no subsidies 
Maryland Did not start until Block None 1 year 
2013 
Massachusetts 1 year Block Licensed centers that received subsidies 2 years 
Licensed centers with no subsidies 
Minnesota First year Hybrid None 2 years 
North Carolina 13 years Points All participating programs 3 years 
Ohio 6 years Hybrid None 1-3 years by rating level 
Rhode Island 3 years Block None 3 years 
Washington First year Hybrid Licensed centers that received subsidies 3 years 


Licensed centers with no subsidies 
Source: Telephone interviews conducted by Mathematica, October 2014-April 2015 and December 2016-January 2017. 
a Rating structure applies to levels beyond the lowest level. All TQRIS have a block structure for the lowest level that requires programs to 
meet all licensing requirements and sometimes additional components. 


e Rating structure. TQRIS use one of three structures to determine a program’s rating level: 
points, hybrid, and building block structures (Figure 1). Six states use points and hybrid 
structures, which can provide programs with flexibility in demonstrating quality because 
they offer multiple ways to earn a higher rating. These structures might promote more 
movement of programs up the rating levels than building block structures, which require 
programs to meet all standards of a given rating. 
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Figure 1. Rating structures in TQRIS 


Building block 
v 4 a 


Programs must meet all 
standards in a level to 
receive that rating. 


Points 


= +3 f a 
Fo wl 


‘"@8 


Programs earn points for 
meeting standards. The total 
number of points determines 


Programs are rated by a 
building-block structure in 
some levels (usually, lower 


the rating. levels) and a points structure in 


others (usually, higher ones). 


Entry requirement at lowest level. Four states required programs to enter at the lowest 
level, instead of the level for which they may have qualified. Overqualified programs at 
the lowest level might move quickly to higher levels. 


Validity period. Depending on the state (and sometimes the rating level), ratings are valid 
from one to three years. Shorter validity periods—and more frequent reassessment of 
programs—could cause more frequent increases, and possibly decreases, in ratings. 


The nine states developed policies to attempt to encourage and support TQRIS participation 
among different types of programs (Table 3). These policies might also have affected the ways 
that programs entered and moved through the ratings, resulting in different distributions of 
programs across the rating levels in each state. The state policies were as follows: 


Participation requirements. Seven states required certain types of programs (for 
example, those that received public funding like state-funded pre-K programs, Head Start 
programs, or licensed centers that received child care subsidies) to participate in the 
TQRIS. Participation of licensed centers that did not receive public funding was voluntary 
in every state but North Carolina, which required all licensed programs to participate in the 
TOQRIS. 


Alternative pathways or automatic ratings. Six states aimed to encourage participation 
by awarding credit (known as offering an alternative pathway) for certain quality 
components (five states) or automatically awarding higher rating levels (one state) to 
programs that met quality standards external to the TQRIS (for example, state-funded pre- 
K, and Head Start, and NAEYC-accredited programs). These policies could have resulted 
in certain programs entering the TQRIS at high rating levels and then not moving much, 
depending on the level at which they entered. When alternative pathways or automatic 
ratings are in place, eligible programs are exempted from part or all of the TQRIS rating 
process in an effort to decrease the burden on programs that must meet multiple sets of 
standards. States often complete a crosswalk between other external quality standards and 
the TQRIS standards to establish comparability when developing these types of policies. 
However, the specific degree of equivalence between the relevant external standards and 
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TQRIS standards is not fully known; equivalence can more readily be established for 
specific components used in alternative pathways than for a full set of components that 
automatically qualifies a program for a high rating level. 


Table 3. TQRIS policies in RTT-ELC Round 1 states 


Tiered 


reimbursement 
Rating level payments for Other 
Participation Alternative Automatic funding children receiving silar-laretel| 
requirements pathways rating requirement subsidies incentives 
California None None None 
Delaware State-funded pre-K None NAEYC, None Any program None 
Head Start, receiving 
state-funded subsidies for low- 
pre-K income children 
Maryland Licensed centers NAEYC, None State-funded Any program All 
receiving subsidies, state-funded pre-Ka receiving participating 
state-funded pre-K pre-K subsidies for low- programs, 
(school- income children except 
based only)? school-based 
programs 
Massachusetts Licensed centers NAEYC, None State-funded None None 
receiving subsidies, Head Start pre-K 
Head Start, 
state-funded pre-K 
Minnesota None NAEYC, None None Any program All 
Head Start, receiving participating 
state-funded subsidies for low- programs 
pre-K income children 
North Carolina All licensed programs; None None Licensed Any program None 
includes both types of centers receiving 
licensed centers and receiving subsidies for low- 
may include Head subsidies, income children 
Start and state- state-funded 
funded pre-K pre-K 
programs in licensed 
settings 
Ohio Licensed centers None None None Any program All 
receiving subsidies, receiving participating 
state-funded pre-K subsidies for low- programs, 
income children except 
school-based 
programs 
Rhode Island Licensed centers NAEYC, None None Any program None 
receiving subsidies, Head Start receiving 
state-funded pre-K subsidies for low- 
income children 
Washington State-funded pre-K Head Start, None None Any program All 
state-funded receiving participating 
pre-K subsidies for low- programs 


income children 


Source: Telephone interviews conducted by Mathematica, October 2014-April 2015 and December 2016-January 2017. 


Note: Blue shaded cells indicate states that have the policy listed in the table column. Gray shaded cells indicate states that do not have 
the policy listed in the table column. 


a These policies took effect in the middle of 2015 in Maryland. 


e Rating level funding requirement. In three states, programs had to attain a certain rating 
level to be eligible for specific types of funding. For example, a program in North Carolina 
must have attained a level 3 rating to receive child care subsidies from the Child Care and 
Development Fund or a level 4 rating to receive state pre-K funds. These policies could 
have also channeled certain programs into higher ratings at TQRIS entry and reduced the 
amount of improvement needed to move up. 
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Tiered reimbursement for child care subsidies. Seven states intended to promote 
participation, particularly among licensed centers that received subsidies, by offering 
higher payments to subsidize serving low-income children if the programs participate in 
TQRIS. The tiered rates might also have promoted advancement by incrementally 
increasing the payments to programs at each higher rating level.+ 


Other financial incentives. Four states intended to promote participation and 
advancement up the rating levels by offering other forms of financial assistance to 
programs that participated in TQRIS, such as quality improvement grants that helped 
programs work toward a higher rating or bonus awards for achieving a higher rating level. 


In addition to these policies, states could try to promote quality improvement by publicizing 
TQRIS ratings. This study did not collect systematic data on how states publicized ratings that 
could be used to interpret the patterns of participation and ratings. 


Below we discuss the patterns in TQRIS participation and ratings among programs in the nine 
Round | states during the RTT-ELC grant period. Throughout the findings, it is important to keep 
the following caveats in mind: 


Rating levels do not reflect a uniform level of quality. Because states use different 
quality components and calculate rating levels differently, programs that receive a 
particular rating level might vary substantially in quality across states, even among states 
that use the same number of rating levels. In addition, within states, programs might 
achieve rating levels in different ways, weakening the link between the levels and certain 
measures of quality. For example, building block rating structures require programs to meet 
all standards within a level to receive a rating. In these structures, a program that missed 
qualifying for the next level based on a single standard might not be that different in quality 
from programs at the next level. In points and hybrid structures, programs that receive a 
given rating level could have met different standards, as long as they received enough 
points. 


Rating levels might not reflect programs’ “true” quality. States use various data 
collection and verification processes to help ensure the reliability of the rating levels, but 
ratings could differ from programs’ “true” quality for several reasons. First, some states 
use alternative pathways or automatic ratings, which exempt programs that meet external 
quality standards from part or all of the TQRIS rating process. It is not fully known whether 
these programs would meet the state-defined standards. Second, some states require 
programs to enter at the lowest level, instead of the level to which they would have 
qualified. In these states, ratings might initially understate some programs’ “true” quality. 
Overqualified programs could move up without improving quality, resulting in movement 
that does not reflect increased quality. Finally, programs could make quality improvements 
without applying for higher ratings; this could cause ratings to understate programs’ 
quality. 


Increasing the number of programs at the highest TQRIS rating levels might not 
necessarily improve children’s access to high quality programs. States could increase 
the number of programs at high rating levels by encouraging existing programs that were 
already high quality to participate in TQRIS. In addition, increased availability of high 
quality programs might not translate into increased access for low-income children due to 
other factors, like affordability. 
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e The findings do not provide evidence that particular policies or program types caused 
the observed patterns in participation and ratings. Throughout the findings, we discuss 
the TQRIS characteristics and policies that might influence the patterns we present, but 
this study is purely descriptive. The nine Round | states differ on many TQRIS 
characteristics and policies as well as on other dimensions, such as the characteristics of 
families seeking child care, the way in which TQRIS publicize the ratings to families, the 
composition of the early learning and development landscape, and the presence of other 
quality initiatives beyond TQRIS. Therefore, although we discuss several potential 
explanations for the patterns we present, other potential explanations might exist for these 
observed patterns. This brief does not describe all of the differences across the states that 
might have contributed to differences in the patterns of program participation or ratings 
that we present. 


Findings 


How did TQRIS participation change during the RTT-ELC grant period from 2012 to 
2016 in the nine states? What was the composition of programs that participated in 
TQRIS by 2016? 


1. TQRIS participation increased across all states and program types during the RTT-ELC 
grant period. 


The goal of the RTT-ELC grants was to help states promote participation in TQRIS among a 
range of early learning and development programs, including those that served low-income 
children. The nine Round | states successfully developed TQRIS and/or expanded participation in 
TQRIS during the grant period, particularly among programs that served low-income children 
including state-funded pre-K programs, Head Start, and licensed centers that received subsidies. 


TQRIS participation increased in each of the nine states during RTT-ELC from 2012 to 
2016, especially in states that implemented TQRIS toward the beginning of the grant period. 
By 2016, six states had more than half of all eligible programs participating in TQRIS, and three 
of these states had participation rates of 80 percent or more (Figure 2). In five states, participation 
rates increased by more than 40 percentage points. Overall, just under half (48 percent) of all 
center-based programs across the states participated in TQRIS by 2016, a 10 percentage point 
increase from 2014 (the earliest year for which data on all states were available). 


The growth in participation varied across the states, 


Policies intended to encourage =» nossibly due, in part, to a combination of TQRIS 
MSHI) Seuulelsc Tel characteristics (such as the length of TQRIS 
e Participation implementation), and policies (such as _ participation 

requirements requirements). North Carolina consistently had the highest 
e Alternative pathways TQRIS participation rate, ranging from 85 percent in 2012 to 
e Automatic ratings 88 percent in 2016. North Carolina’s rating system has been 
e Financial incentives in place since 1999 and the state requires all licensed centers 


to participate. 


California, Maryland, Minnesota, and Washington were just launching their TQRIS when they 
received their RTT-ELC grants in 2012, and they had some of the lowest participation rates when 
the grant period began. Maryland did not officially launch its system until 2013, so it had no 
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programs participating in 2012. California launched its system in 16 counties in 2012, but it did 
not have data available until 2014. 


Figure 2. Percentage of programs participating in TQRIS over the RTT-ELC period, by state, 2012-2016 


100 
90 i ooe~e North Carolina 
Delaware 
80 Rhode Island 
70 
ra Maryland 
© Minnesota 
3 60 Washington 
row 
50 ALL STATES 
s Massachusetts 
5 40 
oO 
2 
30 Ohio 
/ California 
10 
0 
2012 2013 2014 2015 2016 


Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of all center-based programs (from the four program types) in the states ranged from 14,287 in 2012 to 28,355 
in 2016. Analysis counted all licensed centers, Head Start, and state-funded pre-K programs that were considered participating by 
each state, including those that had not yet received a rating. California data are from 16 counties that implemented TQRIS 
beginning in 2012; all other data are statewide. The figure omits California and Ohio before 2014. Data were not available from 
California before 2014. Due to significant system changes in Ohio in late 2013, the analysis included only data from 2014 to 2016. 
Maryland implemented TQRIS in 2013; therefore, no programs were participating in 2012. 

Figure reads: Twelve percent of all early learning and development programs participated in TQRIS in Rhode Island in 2012 and 80 percent 


participated by 2016. 
RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


Seven of the nine states required at least some types of programs to participate, which might 
have contributed to increased participation over time. For example, Rhode Island required 
programs that recetved Child Care and Development Fund subsidies to participate in TQRIS 
beginning in 2014. In that year, many licensed centers that received subsidies entered the system, 
increasing TQRIS participation by over 50 percentage points. 


California and Ohio had the lowest TQRIS participation rates among early learning and 
development programs from 2014 through 2016 (the years that data were available for these 
states). California allowed voluntary TQRIS participation for all programs. In 2014, Ohio required 
TQRIS participation for licensed centers that received Child Care and Development Fund 
subsidies and state-funded pre-K programs. However, it gave programs time to enter the system, 
which might explain why their participation increased slowly. 
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TQRIS participation increased for all types of programs from 2012 to 2016, but 
programs that received public funding to serve low-income children participated in TQRIS 
at higher rates than programs that did not. By 2016, about 90 percent of state-funded pre-K 


and Head Start programs in the seven states that had data 


Participation among NAEYC available for the whole period participated in TQRIS 
accredited programs (Figure 3).4 Participation among licensed centers that 
Accredited programs, which received subsidies also increased over the period, 
include all types, participated in approaching the rates for state-funded pre-K and Head 
TQRIS at increasingly higher rates Start programs. Licensed centers that did not receive 


from 2012 to 2015 and then 
remained flat at about 70 percent 


subsidies had the lowest participation rates; just 28 


in 2016 (see Appendix B, Figure percent of these programs in the seven states participated 
B.1). NAEYC-accredited programs in TQRIS in 2016. 


that received some type of public 
funding (state pre-K, Head Start, or 


These patterns of participation might be due 


subsidy funds) participated at somewhat to states’ policies intended to encourage 
much higher rates than accredited programs to participate in TQRIS. Most of the seven 
centers that did not receive these states require or attempt to make it easier for state-funded 
SERIE UUee: pre-K, Head Start, and licensed centers that receive 


subsidies to participate: 


State-funded pre-K programs. Five of the seven states required state-funded pre-K 
programs to participate. In addition, North Carolina required all licensed programs to 
participate and a substantial majority of its state-funded pre-K programs were licensed. 


Head Start programs. A substantial majority of North Carolina’s Head Start programs 
were licensed, and thus, required to participate. In addition, Massachusetts required Head 
Start programs to participate, and four other states intended to make it easier for Head Start 
programs to participate by offering them either an alternative pathway or an automatic 
rating. 


Licensed centers that received subsidies. Four of the seven states (including North 
Carolina) required licensed centers that received subsidies to participate. In contrast, 
participation is voluntary for licensed centers that did not receive subsidies in every state 
but North Carolina. 


2. The majority of programs participating in TQRIS by 2016 received public funds to serve 
low-income children. 


The RTT-ELC grants were intended to help states promote participation in TQRIS, 
particularly among programs that served low-income children. The prevalence of the different 
types of programs participating in each state provides information about which types of programs 
states could target in an attempt to improve the availability of high quality care for low-income 
children. 
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Figure 3. Percentage of programs participating in TQRIS, by program type, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of all center-based programs (from the four program types) in the seven states included in this analysis ranged 
from 14,287 in 2012 to 14,100 in 2016. To facilitate comparisons across years, analysis included only states with data in all years. 
It excluded California and Ohio. 


Figure reads: Thirteen percent of all licensed centers that did not receive subsidies participated in TQRIS in 2012; 28 percent of licensed 
centers that did not receive subsidies participated by 2016. 


pre-K = prekindergarten; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


Licensed centers that received subsidies accounted for the majority of programs 
participating in the TQRIS in 2016 in seven of the nine RTT-ELC Round 1 states. In the 
seven states, these centers represented 54 to 70 percent of participating programs (Figure 4). In the 
other two states—California and Minnesota—state-funded pre-K programs accounted for the 
majority of TQRIS participating programs. 


The prevalence of each type of participating program might be due to some degree to the 
following factors: 


e Relative prevalence of early learning and development programs operating in each 
state. In the seven states where licensed centers that received subsidies represented the 
majority of participating programs, these centers also represented the largest proportion of 
programs that served low-income children operating in those states, exceeding that for 
state-funded pre-K and Head Start programs combined (Appendix A, Figure A.1). In 
contrast, both California and Minnesota had more state-funded pre-K programs operating 
in the state (or 16 counties in the case of California) than they had licensed centers that 
received subsidies and Head Start programs combined. 


e TQRIS policies that were intended to promote participation of particular types of 
programs. Six of the seven states where licensed centers that received subsidies were in 
the majority (all but Massachusetts) offered financial incentives (tiered reimbursement 
rates) to these programs if they participated in TQRIS; these states paid higher subsidies 
per child to centers that participated than to centers that did not. Massachusetts did not 
offer these financial incentives, but it required licensed centers that received subsidies to 
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participate in the system. In the two states where licensed centers that received subsidies 
were in the minority of participating programs, Minnesota offered these programs financial 
incentives but California did not. In addition, Minnesota—one of the two states in which 
state-funded pre-K programs represented the majority of TQRIS participating programs— 
provided alternative pathways to these programs to encourage them to participate. 


Figure 4. Composition of TQRIS participating programs, by state, 2016 
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m= Licensed centers, received subsidies mLicensed centers, no subsidies Head Start m= State-funded pre-K 
Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of TQRIS-rated programs was 13,357 in 2016. Analysis included licensed centers, Head Start programs, and 
state-funded prekindergarten programs that were considered participating by each state, including those that had not yet received 
a rating. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. 


Figure reads: Of the programs that were rated by TQRIS in 2016 in California, 8 percent were licensed centers that received subsidies, 19 
percent were licensed centers that did not receive subsidies, 21 percent were Head Start programs, and 52 percent were state-funded pre- 
K programs. 


pre-K = prekindergarten; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


How did the percentage of programs in the top TQRIS rating levels change from 2012 
to 2016 in the nine states? Did the distribution of ratings shift toward the higher levels 
over time? 


1. Most states experienced modest increases in the percentage of programs in the top two 
TQRIS rating levels during the analysis period. 


To promote access to high quality programs, RTT-ELC states aimed to increase the number 
and percentage of programs in the top levels of the TQRIS. This discussion focuses on the top two 
rating levels, but findings are similar for the full distribution of ratings across the levels (see 
Appendix B, Figures B.2 to B.10, for ratings distributions by state from 2012 to 2016). 


Most states had modest gains in the percentage of programs in the top two rating levels 
over the study period. The majority of programs across all states were rated at the highest or 
second-highest rating level in each year from 2014 to 2016 (Figure 5). The percentage of programs 
at the highest or second-highest rating level was slightly higher in 2016 than in 2014 (55 and 53 
percent, respectively). 
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However, the nine states had markedly different patterns in the percentages of programs in 
the top levels. In seven of the nine states, the percentage of top-rated programs was higher by 2016 
than it was at the start of the analysis period, but many states did not experience a consistent 
increase over time. The most substantial increase in top-rated programs occurred in California 
from 2014 to 2016, possibly due, in part, to the entry of many state-funded pre-K programs that 
were rated at high levels. 


Two states experienced decreases in the percentage of programs in the top levels over time. 
In Minnesota, the decrease was small, and the state still had the highest percentage of top-rated 
programs among all nine states in 2016. In Rhode Island, the percentage of programs at the top 
levels dropped considerably in 2014, coinciding with an influx of licensed centers that entered at 
low rating levels. The small number of TQRIS participating programs in Rhode Island might have 
made the distribution of programs at the top levels sensitive to any change in the types of programs 
participating. 


Figure 5. Percentage of programs in the top two state-defined rating levels, by state, 2012-2016 


100 
«@ 80 North Carolina 
5 70 Om Delaware 
S 60 California 
a 
S ALL STATES 
o 00 Ohio 
3 
S 30 
o 20 Rhode Island 
Massachusetts 
10 == Washington 
0 _ == = | Maryland 
2012 2013 2014 2015 2016 


Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of programs ranged from 5,414 in 2012 to 13,357 in 2016. Analysis included only programs that were rated. 
Ratings are measured at TQRIS entry and in June of each following year. Rating categories reflect the order of ratings within states’ 
rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. Only the top two state- 
defined rating levels were considered for this analysis. Massachusetts started granting the second highest and highest rating levels 
in January and November 2014, respectively. California data are from 16 counties that implemented TQRIS beginning in 2012; all 
other data are statewide. The figure omits California and Ohio before 2014. Data were not available from California before 2014. 
Due to significant system changes in Ohio in late 2013, the analysis included only data from 2014 to 2016. 


Figure reads: Less than 1 percent of programs in Massachusetts were rated in one of the top two state-defined rating levels in 2012; 14 
percent were rated in the top two levels by 2016. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


How did TQRIS participating programs in the nine states achieve the top two rating 
levels by 2016? Did they enter at the highest rating levels or did they improve to 
achieve higher ratings? 


To increase the number of programs at the top rating levels, RTT-ELC states could either 
encourage programs that were already high quality to enter TQRIS or encourage lower-quality 
programs to participate in TQRIS and motivate them to improve. The latter is a key way that 
TQRIS intend to increase access to high quality programs. This section examines whether 


NCEE EVALUATION BRIEF 


18 MOVING ON UP? PROGRAM QUALITY RATINGS UNDER 
RACE TO THE TOP—EARLY LEARNING CHALLENGE 


programs that entered TQRIS in RTT-ELC states already had high ratings at entry or had low 
ratings at entry and, thus, room for improvement. 


1. Entry ratings varied across programs, with most state-funded pre-K and Head Start 
programs entering at high rating levels and most licensed centers entering at low rating 
levels. 


Entry ratings for different types of programs varied substantially. Forty percent of all 
programs that entered the TQRIS during the study period received one of the two highest rating 
levels and slightly more than half (53 percent) received one of the two lowest ratings (Figure 6). 
This section presents findings by program type, but ratings at entry also varied by state (see 
Appendix B, Figure B.11). 


Most state-funded pre-K and Head Start programs that participated in TQRIS entered 
the system at the highest or second-highest level. Nearly 60 percent of state-funded pre-K and 
Head Start programs entered TQRIS at one of the top two rating levels. 


States’ policies regarding ratings at entry might partially explain these high ratings. For 
example, because state-funded pre-K and Head Start programs meet external quality standards, 
many states automatically award these programs credit for meeting certain TQRIS components or 
rating levels when they enter TQRIS. States could also require these programs to achieve a certain 
rating level to be eligible for funds. Seven states had at least one policy in place that could fast- 
track these types of programs to high rating levels: 


e Three states offered alternative pathways to higher- Characteristics that could 


level ratings to state-funded pre-K programs. influence ratings at entry: 
e Requirement that 


e Four states offered alternative pathways to Head programs must enter at 
Start programs. lowest level 
e Delaware automatically rated both state-funded pre- Policies that could influence 
ratings at entry: 


K and Head Start programs at the second-highest 


level upon TQRIS entry. e Alternative pathways 


E . e Automatic ratings 
e Two states required programs to achieve the second- 


highest or highest level to be eligible to receive state 
pre-K funds. 


e = Rating level funding 
requirement 


However, because state-funded pre-K and Head Start programs meet external quality 
standards, it is possible they would still receive the same rating if they had gone through states’ 
full rating process. Previous research does not clearly indicate whether they would receive the 
same ratings. Some TQRIS studies in RTT-ELC Round | states found that programs that entered 
through the alternative pathway had similar scores on observational measures of quality as 
programs that received a high rating level through the TQRIS rating process (Tout et al. 2016; 
Roberts et al. 2016). However, another study found that state-funded pre-K programs that served 
a majority of low-income children and entered through an alternative pathway had lower 
observational scores than regularly rated programs at the highest levels (Karoly et al. 2016). 


The majority of licensed centers (including those that did and did not receive subsidies) 
entered the TQRIS at the lowest rating level, and thus had room for improvement. This could 
be due partly to states’ entry requirements for licensed centers. Four states required licensed 
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centers to enter the TQRIS at the lowest rating level, unless they met external quality standards 
(such as NAEYC accreditation). However, most licensed centers participating in TQRIS did not 
have NAEYC accreditation. 


Another potential explanation for licensed centers receiving lower ratings at TQRIS entry than 
Head Start and state funded pre-K programs is the age range of the children they serve. Licensed 
centers often serve infants and toddlers in addition to preschool-age children. Meeting standards 
for infants and toddlers might be more challenging because of requirements such as smaller group 
sizes and those associated with diapering and other health and safety issues. For example, a study 
of potential TQRIS designs in California found that center-based classrooms serving infants and 
toddlers received lower scores on observational measures of quality than classrooms serving 
preschool-age children (Karoly and Zellman 2012). 


Figure 6. Percentage of programs that attained each state-defined rating level at TQRIS entry, by 
program type, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 9,658. Analysis included only programs that entered between 2012 and 2016. Rating categories 
reflect the order of ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having 
no middle level. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The 
figure omits California and Ohio before 2014. Data were not available from California before 2014. Due to significant system 
changes in Ohio in late 2013, the analysis included only data from 2014 to 2016. 


Figure reads: Eleven percent of state-funded pre-K programs entered TQRIS at the lowest state-defined rating level, 7 percent at the second 
lowest level, 6 percent at the middle level, 17 percent at the second-highest level, and 59 percent at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


2. Many programs remained at the same level during the study period, but some moved up 
from lower ratings over time. 


States aim to increase the availability of high quality programs for low-income children by 
moving programs from low to high ratings. The degree of improvement possible within the study 
period depended on the ratings that programs received when they entered TQRIS and how much 
time they had to improve. This analysis provides a picture of movement among all programs that 
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participated during the analysis period for each state, regardless of when the programs entered (or 
exited) the TQRIS. 


Characteristics that could 
influence ratings improvement: 


e Rating structure 


e Requirement that programs 
must enter at lowest level 


e Validity period for ratings 


Policies that could promote 


ratings improvement: 


e Financial incentives, such 
as quality improvement 
grants or tiered 
reimbursement rates 


Most programs remained at the same rating level 
during the study period, but some increased their rating 
level over time. Over half of programs were in one of the 
three lowest rating levels at the beginning of the study 
period. Eleven percent of programs were able to achieve one 
of the top two levels by improving their rating during the 
study period (Figure 7, dark blue bubbles in outlined box). 
Another 44 percent of programs had a first and last rating at 
the highest or second-highest level; the majority of these (37 
percent of all programs) retained the same rating over this 
period (the two highest gray bubbles in Figure 7). Like 
programs’ ratings at entry, patterns of movement to higher 
ratings differed across program types. 


Figure 7. Changes in programs’ state-defined ratings over the study period, all program types, 2012- 
2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Notes: 


The total number of programs was 14,042. Figure includes all programs that participated in TQRIS in the nine states for any period 
from 2012 through 2016 (except those that entered in 2016) and examines the change in ratings from the first rating in the 
analysis period to the last rating in the analysis period. Rating level changes are reported for the full period of 2012 to 2016 for 
seven states, and 2014 to 2016 for California and Ohio. California data are from 16 counties that implemented TQRIS beginning 
in 2012; all other data are statewide. Rating categories reflect the order of ratings within states’ rating structure, and treat states 
with four levels (Massachusetts and Minnesota) as having no middle level. 


Figure reads: Fourteen percent of programs were at the lowest state-defined rating level at the beginning of the study period and remained 
at that rating for the entire study period. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Over one-quarter of licensed centers that received subsidies achieved higher ratings over 
the period, but most did not end up at the top rating levels. Forty-four percent of licensed 
centers that received subsidies were at one of the three lower rating levels at the beginning of the 
study period, and remained at the same level over this period (the three lowest gray bubbles in 
Figure 8). Some centers might not have had time to improve by multiple levels, or at all, if they 
entered later in the study period. One in 10 licensed centers that received subsidies was able to 
achieve one of the two highest levels by improving their rating over the period (the dark blue 
bubbles in the outlined box). Another 4 percent improved from the second-highest to the highest 
rating level. Thirteen percent improved from the lowest or second-lowest rating but did not reach 
one of the top levels. (Licensed centers that did not receive subsidies had similar patterns of 
movement, presented in Appendix B, Figure B.12.) 


Figure 8. Changes in programs’ state-defined ratings over the study period, licensed centers that 
received subsidies, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Notes: The total number of programs was 6,740. The figure includes all licensed centers that received subsidies that participated in TQRIS 
in the nine states for any period from 2012 through 2016 (except those that entered in 2016) and examines the change in ratings 
from the first rating in the analysis period to the last rating in the analysis period. Rating level changes are reported for the full 
period of 2012 to 2016 for seven states, and 2014 to 2016 for California and Ohio. California data are from 16 counties that 
implemented TQRIS beginning in 2012; all other data are statewide. Rating categories reflect the order of ratings within states’ 
rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. 


Figure reads: Twenty percent of licensed centers that received subsidies were at the lowest state-defined rating level at the beginning of the 
study period and remained at that rating for the entire study period. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Nearly one in three Head Start programs moved to a higher rating level over the period, 
even though many were in the top ratings to start. Half of Head Start programs were at one of 
the two highest levels at the beginning of the study period, and remained at the same level over 
this period (the two highest gray bubbles in Figure 9). Fourteen percent improved to achieve one 
of the two highest rating levels (the dark blue bubbles in the outlined box). Another 11 percent 
improved from the second-highest to the highest level. 


Figure 9. Changes in state-defined ratings over the study period, Head Start programs, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 

Notes: The total number of programs was 1,566. The figure includes all Head Start programs that participated in TQRIS in the nine states 
for any period from 2012 through 2016 (except those that entered in 2016) and examines the change in ratings from the first 
rating in the analysis period to the last rating in the analysis period. Rating level changes are reported for the full period of 2012 
to 2016 for seven states, and 2014 to 2016 for California and Ohio. California data are from 16 counties that implemented TQRIS 
beginning in 2012; all other data are statewide. Rating categories reflect the order of ratings within states’ rating structure, and 
treat states with four levels (Massachusetts and Minnesota) as having no middle level. 

Figure reads: Three percent of Head Start programs were at the lowest state-defined rating level at the beginning of the study period and 

remained at that rating for the entire study period. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


State-funded pre-K programs experienced the least movement over the study period 
because they had high ratings to start. About 21 percent of state-funded pre-K programs 
achieved a higher rating level over the study period (Appendix B, Figure B.13). The majority of 
them had a rating at the very highest level at the beginning of the study period and stayed at that 
level (56 percent). Another 12 percent had a rating at the second-highest level at the beginning of 
the study period and stayed there. 
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3. Percentages of top-rated programs by 2016 and how they reached that level differed 
substantially by state. 


The analysis presented above examines entry and movement among all programs that 
participated in TQRIS during the RTT-ELC period. In this section, we focus on how the set of 
programs that received ratings in the top two levels in 2016—the end of the RTT-ELC period— 
had attained those ratings. Among programs that received top ratings at entry, we distinguish 
between programs that were eligible for an alternative pathway or automatic rating based on their 
state’s policies and those that were not. 


States differed in the percentages of programs that achieved ratings in the top two levels 
by 2016 and how programs reached those levels. For example, in Minnesota, nearly all programs 
participating in TQRIS received one of the two highest rating levels in 2016 and had achieved 
those ratings at entry (Figure 10). North Carolina also had a substantial majority of programs reach 
top rating levels by 2016, but most of its programs already had these ratings at the start of the 
analysis period in 2012. A majority of programs also achieved top rating levels in Delaware, 
California, and Ohio, but roughly equal portions in those states achieved these ratings through high 
ratings at entry and ratings improvement. In contrast, few programs in Maryland, Massachusetts, 
Rhode Island, and Washington were at top rating levels by 2016. 


Several TQRIS characteristics and policies might have contributed, in part, to these patterns: 


e Length of TQRIS implementation. North Carolina had the longest-running TQRIS of the 
nine states, which could have given programs more time to reach top rating levels and is 
one possible explanation for why so many programs had already reached those levels by 
2012. In contrast, two of the states with the lowest percentages of programs at top rating 
levels (Washington and Maryland) were just launching their TQRIS when they received 
their RTT-ELC grants in 2012. Massachusetts also had a relatively new TQRIS and did not 
start granting the two highest rating levels until 2014, in the middle of the study period. 


e Rating structure. The three states with building block rating structures (Maryland, 
Massachusetts, and Rhode Island) had low percentages of programs that achieved top 
rating levels at entry or through ratings improvement. One possible explanation for this is 
that building block rating structures might make movement up the ratings more challenging 
because programs must meet all standards at each rating level before progressing to the 
next level. Hybrid and points rating structures might offer programs greater flexibility than 
building block structures in meeting requirements for the higher levels. 


e Validity period for ratings. Delaware, Ohio, and Maryland had the shortest validity 
periods (one year for programs in Maryland and one year for programs in the low rating 
levels in Delaware and Ohio). This is one possible explanation for why ratings in Delaware 
and Ohio improved in these states—lower-rated programs might have reapplied for ratings 
relatively quickly. 


e Alternative pathways and automatic ratings. Minnesota was the only state that offered 
an alternative TQRIS pathway to all programs that meet external quality standards 
(including state-funded pre-K, Head Start, and NAEYC-accredited programs). The vast 
majority of programs that achieved one of the two highest rating levels in Minnesota were 
eligible for the alternative pathway. In contrast, in other states, lower percentages of 
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programs that reached top rating levels were eligible for either alternative pathways or 
automatic ratings. 


e Rating level funding requirement. North Carolina’s requirement that licensed centers 
that received subsidies and state-funded pre-K programs meet minimum rating levels—the 
middle level and second-highest level, respectively—to receive public funding might have 
contributed in some way to its high percentages 
of programs that reached top rating levels. These 


Characteristics that could influence 


types of programs represented the substantial attainment of the top rating levels: 
majority of TQRIS participating programs in e Length of TQRIS implementation 
North Carolina in 2016. Massachusetts and ° Rating structure (building block, 
Maryland also had these types of policies, but points, or hybrid) 

licensed centers, not state-funded pre-K © Validity period for ratings 


programs, represent the majority of participating 


programs in those states Policies that could promote 


attainment of the top rating levels: 


e Financial incentives. All of the states except e Alternative pathways 
two (California and Massachusetts) offer tiered e Automatic ratings 
reimbursement rates, which increase the e Rating level funding requirement 
subsidies programs receive for serving low- e Financial incentives, such as 
income children as programs attain higher rating quality improvement grants or 
levels. These subsidies might contribute, at least tiered reimbursement rates 


partially, to ratings improvement, particularly 
among licensed centers that receive subsidies, which enter at low ratings. 


The nine Round | states also differ on many other dimensions, such as the composition of 
programs in each state, the way in which TQRIS publicize ratings to families, the characteristics 
of families seeking child care, and the presence of other quality initiatives beyond TQRIS. 
Therefore, other potential explanations (beyond those discussed here) for the observed patterns are 
possible. 


Did states make progress in promoting availability of high quality programs for low- 
income children? 


To increase the number and percentage of children with high needs who attend programs in 
the top rating levels, states could try to expand participation of programs that serve these children 
and help them improve. Children with high needs could include those from low-income families, 
as well as those who have disabilities or developmental delays, are English learners, or are migrant, 
homeless, or in foster care, in addition to other factors the states identified. To examine the 
percentage of high-needs children who attended programs in top rating levels, we focused on the 
number of children in licensed centers that received subsidies (some of these children came from 
low-income households).° 


1. States did not consistently provide information on the number of children with high 
needs who attend each TQRIS-participating center. 


RTT-ELC aimed to support integration of state early learning and development systems, but 
the data still reside with many different agencies. States provided complete data on the total 
number of children a program could serve (capacity) only for licensed programs. States could not 
consistently provide details on the number of children served (based on enrollment or capacity 
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data) in Head Start and state-funded pre-K programs that were exempt from child care licensing. 
These data were not (1) collected or maintained by the states at the center level, (2) collected 
centrally, or (3) maintained historically. In addition, they could not be made readily available to 
merge with other data collected at the center level. 


Figure 10. Percentages of programs that achieved the highest or second-highest state-defined TQRIS 
rating level in 2016 and how they got there, by state 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 13,357. Analysis was based on all programs that achieved the highest or second-highest rating 
level in each state in 2016. “Achieved by start of analysis period” includes programs that were at the highest or second-highest 
rating at the beginning of the study period (in 2014 for California and Ohio and in 2012 for the other seven states) and for which 
we cannot determine whether they had achieved those levels at entry. California data are from 16 counties that implemented 
TQRIS beginning in 2012; all other data are statewide. 

Figure reads: Ninety-five percent of all programs that participated in TQRIS in 2016 in Minnesota were at the highest or second-highest rating 

level; 11 percent of programs were not eligible for an automatic or alternate pathway and received the highest or second-highest rating level 

at entry, 80 percent were eligible for one of these policies and received the highest or second-highest rating level at entry, and an additional 

3 percent achieved the highest level through upward movement. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


Only a few states could provide complete data on the number of low-income children 
supported through child care subsidies in a center-based program. None of the states could provide 
data at the center level on the number of children supported with either Head Start or state pre-K 
funds. The lack of data prevents looking longitudinally at patterns in the number of low-income 
children in TQRIS-participating programs and in programs that achieve the top rating levels. 


2. States might have increased the availability of high quality programs for low-income 
children by using policies intended to promote participation among licensed centers that 
received subsidies. 


For all children who attended licensed centers that received subsidies, we examined the 
percentages who were in TQRIS-participating centers during the analysis period and the 
percentage in centers in the top rating levels in 2016. 
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States increased the percentage of children served in licensed centers that received 
subsidies that were in TQRIS-participating centers during the analysis period. By 2016, in 
every state except Ohio, the majority of children served by licensed centers that received subsidies 
were in centers that participated in TQRIS, mirroring the growth in TQRIS participation among 
these centers (discussed earlier). Across all states, the percentage of children in licensed centers 
that received subsidies in TQRIS-participating programs rose from 56 percent in 2014 to 70 
percent in 2016 (Figure 11). 


Figure 11. Percentages of children served in licensed centers that received subsidies that participated 
in TQRIS, by state, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of all licensed centers that received subsidies in the states for which the analysis calculated the number of 
children ranged from 6,935 in 2012 to 9,210 in 2016. Analysis is based on licensed capacity for licensed centers that received 
subsidies with a TQRIS rating level in each year. California data are from 16 counties that implemented TQRIS beginning in 2012; 
all other data are statewide. The figure omits California and Ohio before 2014. Data were not available from California before 2014. 
Due to significant system changes in Ohio in late 2013, the analysis included only data from 2014 to 2016. Maryland implemented 
TQRIS in 2013; therefore, no programs were participating in 2012. 


Figure reads: Fourteen percent of all children served in licensed centers that received subsidies in 2012 in Rhode Island were in a center 
that participated in TQRIS; 100 percent of these children were in a center that participated in TQRIS by 2016. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Most children served by licensed centers that received subsidies were not in programs 
at the top rating levels by 2016. Most children served in licensed centers that received subsidies 
were in TQRIS participating centers by 2016; however, 30 percent of all children served in these 
centers were in top-rated centers across the states (Figure 12). In Minnesota, North Carolina, and 
Delaware, 60 percent or more of all children served in licensed centers that received subsidies were 
in programs at the highest TQRIS rating levels in 2016. However, in five states, less than 15 
percent of all children in licensed centers that received subsidies were in centers that had reached 
the top rating levels. 


Figure 12. Percentages of children served in licensed centers that received subsidies that participated 
in TQRIS and those served in centers that had attained one of the top state-defined rating levels in 
2016, by state 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of all licensed centers that received subsidies in 2016 for which the analysis calculated the number of children 
was 8,852. Analysis is based on licensed capacity for licensed centers that received subsidies and, for those participating in TQRIS, 
had a rating level in 2016. Some centers in California, Massachusetts, and Washington are considered participating but are not 
rated and were excluded from this analysis. California data are from 16 counties that implemented TQRIS beginning in 2012; all 
other data are statewide. 


Figure reads: Forty-one percent of all children served in licensed centers that received subsidies in 2016 in Ohio were in a center that 
participated in TQRIS; 12 percent were in a center that had received the highest or second-highest state-defined rating level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


These findings suggest that states might be able to increase the numbers of licensed centers 
that receive subsidies that attain high ratings. Some states have high rates of TQRIS participation 
for these centers, but small percentages of these centers in the top two rating levels. For example, 
in Rhode Island, nearly all children who attended licensed centers that received subsidies in 2016 
were in programs participating in TQRIS. However, 11 percent of these children were in programs 
at one of the top rating levels. In contrast, other states have lower rates of TQRIS participation 
among centers that serve low-income children. For example, in Ohio, just over 40 percent of 
children served by licensed centers that received subsidies were in programs participating in 
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TQRIS in 2016, yet about the same percentage as Rhode Island—12 percent—were in programs 
rated at one of the top levels. Encouraging licensed centers to achieve higher ratings is therefore 
one possible way to expand availability of high quality programs for low-income children. 


Conclusion 


Overall, the nine states were able to increase the number of top-rated programs during the 
RTT-ELC period in two ways. First, states expanded TQRIS participation among state-funded pre- 
K and Head Start programs. These programs entered at high rating levels because they met states’ 
TQRIS standards for these rating levels, or because, before they entered TQRIS, they met external 
quality standards that states used as proxies for meeting certain components or rating levels. 
Second, the nine states moved licensed centers that received subsidies from lower to higher rating 
levels over time. However, in 2016, most children served by licensed centers that received 
subsidies still were not in centers that received top rating levels. This suggests that states might be 
able to continue to expand the availability of high quality programs for low-income children by 
continuing to encourage quality improvements in these programs. 


As states continue to develop and refine their TQRIS, several avenues of additional research 
could further inform policy and practice. First, this brief found that states could not provide data 
on the number of children with high needs that programs serve. Collecting these data consistently 
at the center level across the different types of funding sources could help states better monitor 
progress toward increasing the number of children with high needs served by programs in the top 
rating levels. This could also facilitate additional research aimed at increasing children’s access to 
high quality programs. Second, the analyses presented in this brief aggregate the data across the 
nine states to describe patterns of participation and movement by program type; there may also be 
important differences that exist between the states. 


Third, this brief discusses TQRIS characteristics and policies that might partially contribute 
to the observed patterns, but it does not isolate the effect of any TQRIS characteristic or policy on 
programs’ participation or ratings. Therefore, additional research is needed to provide evidence on 
the effects of these TQRIS characteristics and policies on programs’ outcomes. This research 
might include studies that use research designs that randomly vary the incentives, technical 
assistance, or professional development that programs received. 


Finally, this brief examines whether programs attained higher ratings, but higher ratings might 
not improve outcomes for children if these ratings do not convey meaningful differences in 
program quality. One report that synthesized TQRIS validation studies found that higher-rated 
programs had higher scores on observational measures of quality than lower-rated programs, but 
there was limited evidence that differences in quality translated to differences in children’s 
outcomes (Tout et al. 2017). As TQRIS continue to be refined, research could continue to examine 
the relationship between TQRIS ratings and children’s outcomes. 
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Endnotes 


' For more information on the history and implementation of TQRIS, see the report from this study, Kirby et al. (2017). 


? Information from BUILD Initiative (2017) on the current 44 TQRIS indicates a wide range in characteristics such as 
rating structure and validity period. For example, 18 TQRIS use a hybrid rating structure, 19 a building block, 
and 7 a points structure. The validity period is three years in 17 TQRIS, two years in 9, one year in 8, and a 
period that falls outside these categories in 10 TQRIS. 


3 Rhode Island did not call its incentives tiered reimbursement but they functioned in the same way and we classified 
the policy as such for cross-state aggregation. 


4 Both California and Ohio had increases in participation rates for all program types from 2014 to 2016. The pattern 
of participation among program types in California was similar to that of the other states. In Ohio, Head Start 


programs had the highest participation rates, more than double that of all the other types. 


> Not all states could provide data on the specific number of subsidized children served in each center, so the analysis 
focused on total capacity of these centers. 
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APPENDIX A. METHODS AND SAMPLE SIZES 


State administrative data sources. The administrative data we collected came from multiple 
sources within each state. The landscape of center-based early learning and development programs 
that serve young children is quite diverse, and states have typically decentralized program 
oversight. For example, often distinct offices with separate data systems administer licensing and 
child care subsidies but both are generally under the auspices of a state’s human or social services 
agency, state-funded pre-K programs are generally under the state’s department of education, and 
Head Start is a federally administered program. The RTT-ELC grants aimed, in part, to support 
the integration of state early care and education systems, but the data still reside in many different 
agencies and entities. As a result, the data we requested came from multiple sources within each 
state. In five states (Maryland, Massachusetts, Minnesota, North Carolina, and Washington), we 
worked with a main contact in one department who coordinated with others in the state to gather 
and submit the data. In the four other states (California, Delaware, Ohio, and Rhode Island), we 
obtained data from a variety of sources. 


Standardizing variables across states. We used information from multiple sources to 
maximize our ability to determine licensing status, funding sources, and program closings and to 
build a longitudinal analysis file for each state. To analyze results across states, we used a 
consistent method to construct variables from the raw data each state provided. Our definitions of 
program types or other program elements may not exactly match each state’s definition. As a 
result, the numbers in this brief may differ from those in the annual progress reports that states 
submitted to the U.S. Department of Education for RTT-ELC. The data from the state RTT-ELC 
yearly reports are difficult to assess across states, given the differences in data coverage of children 
and programs, the methods of classifying programs by type, and the methods of counting programs 
in rating levels. In addition, because those data represent a single point in time (that is, they are 
not longitudinal), it is challenging to discern patterns of TQRIS entry and movement up the rating 
levels by specific program types over time. 


Sample sizes. We collected information about all early learning and development programs 
in the state—both those that participated in the TQRIS and those that did not—to provide a 
complete picture of how availability to high quality programs may have changed under RTT-ELC. 
The number of early learning and development programs varies across the states and, therefore, 
influences the number of programs that can participate in the TQRIS. Some states count programs 
as participating in TQRIS before they officially receive a rating. To determine participation rates, 
we counted programs as participating in the same way that each state did. For analyses on ratings 
distribution and movement, we included only programs that had received a TQRIS rating. Table 
A.1 presents the numbers of all early learning and development programs, the number of TQRIS 
participating programs, and the number of TQRIS-rated programs for each year in the analysis 
period by state. 
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Table A.1. Total number of early learning and development programs in the nine Round 1 RTT-ELC states 
and number of TQRIS participating and rated programs, 2012 to 2016 


State and number of programs 2012 2013 2014 2015 2016 
California 

Total na. na. 7,616 8,406 8,442 
Participating na. na. 1,214 2,009 2,226 
Rated na. n.a. 949 1,913 2,153 
Delaware 

Total 373 426 452 455 451 
Participating 123 253 362 378 382 
Rated 123 253 362 377 379 
Maryland 

Total 1,947 1,934 1,922 1,920 1,919 
Participating 0 102 447 1,011 1,266 
Rated 0 102 447 1,002 1,257 
Massachusetts 

Total 2,384 2,713 2,679 2,795 2,597 
Participating 814 1,019 1,103 1,215 1,175 
Rated 795 986 1,072 1,166 1,125 
Minnesota 

Total 2,851 2,851 2,851 2,851 2,851 
Participating 485 1,312 1,586 1,836 1,844 
Rated 485 1,312 1,584 1,827 1,831 
North Carolina 

Total 4,351 4,401 4,355 4,300 4,116 
Participating 3,713 3,756 3,699 3,738 3,629 
Rated 3,651 3,699 3,652 3,696 3,594 
Ohio 

Total n.a. n.a. 6,085 6,024 5,813 
Participating na. na. 1,141 1,561 1,739 
Rated n.a. n.a. 1,128 1,534 1,722 
Rhode Island 

Total 326 332 334 335 330 
Participating 40 55 237 265 264 
Rated 40 55 237 265 261 
Washington 

Total 2,055 1,994 1,932 1,957 1,836 
Participating 356 815 967 1,035 1,089 
Rated 320 773 924 987 1,035 
All states except California and Ohio 

Total 14,287 14,651 14,525 14,613 14,100 
Participating 5,531 7,312 8,401 9,478 9,649 
Rated 5,414 7,180 8,278 9,320 9,482 
All states including California and Ohio 

Total na. na. 28,226 29,043 28,355 
Participating na. n.a. 10,756 13,048 13,614 
Rated n.a. n.a. 10,355 12,767 13,357 


Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: Analysis counted all licensed centers, Head Start, and state-funded pre-K programs in each state. A program was considered 
participating if they were in the TQRIS at any point during the year. Rated programs were analyzed as of June of each year, therefore 
numbers of rated programs may differ slightly than number of participating programs even for states that do not generally have 
unrated programs. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. 
The table omits California and Ohio before 2014. Data were not available from California before 2014. Due to significant system 
changes in Ohio in late 2013, the analysis included only data from 2014 to 2016. Data for 2016 are from January to June. 


n.a. = not applicable; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement system. 


Defining program types. To support cross-state aggregation, we developed mutually 
exclusive program types to mirror those typically found in the state RTT-ELC yearly reports as 


follows: 


e State-funded pre-K.* Includes (1) programs in a school-based setting that received state 
pre-K funds; (2) programs that received only state pre-K funds, or a combination of state 
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pre-K and Child Care and Development Fund or other child care subsidy funds, but not 
Head Start; and (3) a small percentage of licensed centers that received both Head Start 
and state pre-K funds but did not have Head Start in the program name. 


e Head Start. Includes (1) programs that had Head Start in their name, but also may have 
received Child Care and Development Fund or other child care subsidy funds, or state pre- 
K funding; and (2) programs that did not have Head Start in their name, but received only 
Head Start funding, or that recetved Head Start and Child Care and Development Fund or 
other child care subsidy funds. 


e Licensed centers, received subsidies. Includes licensed centers that received Child Care 
and Development Fund or similar subsidies to support low-income children in the program 
but did not receive Head Start or state pre-K funding. 


e Licensed centers, no subsidies. Includes licensed centers that did not receive Child Care 
and Development Fund or other child care subsidy, Head Start, or state pre-K funding. 


Table A.2 presents the percentage of programs within each program type as well as the 
percentages, by program type, that were licensed, accredited, and received the various sources of 
public funding. 


Table A.2. Characteristics of TQRIS participating programs in 2016, by program type 


Percentage of programs 
receiving each type of funding 


Funding 
Number Percentage Percentage eile} Head State Lice)an) 
of of NAEYC Percentage care Start pre-K multiple 
Program type programs programs accredited licensed subsidies funds funds sources 


All programs participating in TQRIS 13,614 100 5 87 57 13 31 13 
State-funded pre-K 3,920 29 2 60 28 6 100 31 

School-based pre-K 2,174 16 1 28 5 2 100 6 

Licensed pre-K 1,746 13 4 100 55 12 100 61 
Head Start 1,468 141 5 90 26 100 19 40 
Licensed, received subsidies 6,325 46 8 100 100 0 0 0 
Licensed, no subsidies 1,901 14 4 100 0 0 0 0 


Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — This analysis included all state-funded pre-K, Head Start, and licensed centers that each state considered participating in 2016, 
including those that had not yet received a rating. School-based pre-K includes a small number of license-exempt programs 
receiving state pre-K funding. The analysis excluded all other license-exempt programs. California data are from 16 counties that 
implemented TQRIS beginning in 2012; all other data are statewide. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement system; NAEYC = National Association 
for the Education of Young Children. 
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Figure A.1. Composition of operating programs, by state, 2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of operating programs was 28,355 in 2016. Analysis included licensed centers, Head Start programs, and state- 
funded prekindergarten programs that were considered operating by each state. California data are from 16 counties that 
implemented TQRIS beginning in 2012; all other data are statewide. 


Figure reads: Of the programs that were operating in 2016 in California, 2.4 percent were licensed centers that received subsidies, 65.5 
percent were licensed centers that did not receive subsidies, 9.3 percent were Head Start programs, and 22.9 percent were state-funded 
pre-K programs. 


pre-K = prekindergarten; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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APPENDIX B. ADDITIONAL FINDINGS 


This appendix presents additional findings on participation and ratings. Figure B.1 presents 
participation rates of accredited programs. Figures B.2 through B.10 present the distribution of 
ratings in each state. Figure B.11 presents ratings at entry by state and Figures B.12 and B.13 
present movement up the rating levels by program type. 


Figure B.1. Percentage of NAEYC-accredited centers participating in TQRIS, 2012-2016 
100 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of programs ranged from 5,531 in 2012 to 9,649 in 2016.To facilitate comparisons across years, analysis 
included only states with data in all years. It excluded California and Ohio. 


Figure reads: Thirty-four percent of all accredited programs participated in TQRIS in 2012; 71 percent participated by 2016. 


NAEYC = National Association for the Education of Young Children; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered 
quality rating and improvement systems. 
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Figure B.2. Distribution of programs’ ratings in California, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs ranged from 949 in 2014 to 2,153 in 2016. Analysis included only programs in California that were 
rated. Ratings were measured at TQRIS entry and in June of each following year. California data are from 16 counties that 
implemented TQRIS beginning in 2012; all other data are statewide. 


Figure reads: Of the programs that participated in TQRIS in 2014 in California, 6 percent were at the state-defined lowest level, 14 percent 


were at the second lowest level, 39 percent were at the middle level, 36 percent were at the second-highest level, and 5 percent were at the 
highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.3. Distribution of programs’ ratings in Delaware, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of programs ranged from 123 in 2012 to 379 in 2016. Analysis included only programs in Delaware that were 
rated. Ratings were measured at TQRIS entry and in June of each following year. 


Percentage of prgorams at each level 


Figure reads: Of the programs that participated in TQRIS in 2012 in Delaware, 2 percent were at the state-defined lowest level, 29 percent 
were at the second lowest level, 8 percent were at the middle level, 43 percent were at the second-highest level, and 17 percent were at the 
highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


Figure B.4. Distribution of programs’ ratings in Maryland, 2012- 2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of programs ranged from O in 2012 to 1,257 in 2016. Analysis included only programs in Maryland that were 
rated. Ratings were measured at TQRIS entry and in June of each following year. 

Figure reads: Maryland did not have any programs participating in TQRIS in 2012. Of the programs that participated in TQRIS in 2013 in 

Maryland, 80 percent were at the state-defined lowest level, 10 percent were at the second lowest level, 2 percent were at the middle level, 

none were at the second-highest level, and 8 percent were at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.5. Distribution of programs’ ratings in Massachusetts, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 

Note: The total number of programs ranged from 795 in 2012 to 1,166 in 2015.The total number was 1,125 in 2016. Analysis included 
only programs in Massachusetts that were rated. Ratings were measured at TQRIS entry and in June of each following year. 
Massachusetts has four rating levels and was considered to have no middle level. Massachusetts started granting the second 
highest and highest rating levels in January and November 2014, respectively. 


Figure reads: Of the programs that participated in TQRIS in 2012 in Massachusetts, 75 percent were at the state-defined lowest level, 24 
percent were at the second lowest level, none were at the middle level, and 1 percent were at the second-highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


Figure B.6. Distribution of programs’ ratings in Minnesota, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 
Note: — The total number of programs ranged from 485 in 2012 to 1,831 in 2016. Analysis included only programs in Minnesota that were 


rated. Ratings were measured at TQRIS entry and in June of each following year. Minnesota has four rating levels and was 
considered to have no middle level. 


Percentage of prgorams at each level 


Figure reads: Of the programs that participated in TQRIS in 2012 in Minnesota, 1 percent were at the state-defined second lowest level, and 
99 percent were at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.7. Distribution of programs’ ratings in North Carolina, 20 12-2016 


100 
90 
State-defined 

80 rating level 
70 # Highest level 

= Second highest level 
60 g 

= Middle level 
50 

Second lowest level 

40 

= Lowest level 
30 
20 
10 

aaa << as Ss 


0 =a 
2012 2013 2014 2015 2016 
Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs ranged from 3,699 in 2013 to 3,594 in 2016. The total number was 3,651 in 2012. Analysis included 
only programs in North Carolina that were rated. Ratings were measured at TQRIS entry and in June of each following year. 


Percentage of prgorams at each level 


Figure reads: Of the programs that participated in TQRIS in 2012 in North Carolina, 4 percent were at the state-defined lowest level, 3 
percent were at the second lowest level, 24 percent were at the middle level, 30 percent were at the second-highest level, and 39 percent 
were at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


Figure B.8. Distribution of programs’ ratings in Ohio, 2012-2016 


100 
State-defined 
rating level 

@ Highest level 

= Second highest level 
= Middle level 

Second lowest level 

= Lowest level 


2012 2013 2014 2015 2016 
Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of programs ranged from 1,128 in 2014 to 1,722 in 2016. Analysis included only programs in Ohio that were 
rated. Ratings were measured at TQRIS entry and in June of each following year. 
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Figure reads: Of the programs that participated in TQRIS in 2014 in Ohio, 11 percent were at the state-defined lowest level, 9 percent were 
at the second lowest level, 38 percent were at the middle level, 29 percent were at the second-highest level, and 13 percent were at the 
highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.9. Distribution of programs’ ratings in Rhode Island, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of programs ranged from 40 in 2012 to 265 in 2015. The total number was 261 in 2016. Analysis included only 
programs in Rhode Island that were rated. Ratings were measured at TQRIS entry and in June of each following year. 


Figure reads: Of the programs that participated in TQRIS in 2012 in Rhode Island, 8 percent were at the state-defined lowest level, 15 
percent were at the second lowest level, 13 percent were at the middle level, 40 percent were at the second-highest level, and 25 percent 
were at the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 


Figure B.10. Distribution of programs’ ratings in Washington, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: — The total number of programs ranged from 320 in 2012 to 1,035 in 2016. Analysis included only programs in Washington that 
were rated. Ratings were measured at TQRIS entry and in June of each following year. 


Percentage of prgorams at each level 


Figure reads: Of the programs that participated in TQRIS in 2012 in Washington, 100 percent were at the state-defined lowest level. 
RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.11. Distribution of program ratings at TQRIS entry from 2012-2016, by state 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Note: The total number of programs was 9,658. Analysis included only programs that entered between 2012 and 2016. Rating categories 
reflect the order of ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having 
no middle level. California data are from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. The 
figure omits California and Ohio before 2014. Data were not available from California before 2014. Due to significant system 
changes in Ohio in late 2013, the analysis included only data from 2014 to 2016. 


Figure reads: In Minnesota, the percentages of programs that entered TQRIS at each state-defined rating level were 2 percent for the lowest 
level, 6 percent for the second lowest level, 1 percent for the second-highest level, and 91 percent for the highest level. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.12. Changes in state-defined ratings over the study period, licensed centers that did not 
receive subsidies, 2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Notes: The total number of programs was 2,127. Figure includes all licensed centers that did not receive subsidies that participated in 
TQRIS in the nine states for any period from 2012 through 2016 (except those that entered in 2016) and examines the change in 
ratings from the first observed rating in the analysis period to the last observed rating in the analysis period. Rating level changes 
are reported for the full period of 2012 to 2016 for seven states, and 2014 to 2016 for California and Ohio. California data are 
from 16 counties that implemented TQRIS beginning in 2012; all other data are statewide. Rating categories reflect the order of 
ratings within states’ rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. 


Figure reads: Twenty-three percent of licensed centers that did not receive subsidies were first observed at the lowest rating level and 
remained at that rating for the entire study period. 


RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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Figure B.13. Changes in state-defined ratings over the study period, state-funded pre-K programs, 
2012-2016 
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Source: Administrative data collected from Round 1 RTT-ELC states from October 2016 to April 2017. 


Notes: The total number of programs was 3,609. Figure includes all state-funded pre-K programs that participated in TQRIS in the nine 
states for any period from 2012 through 2016 (except those that entered in 2016) and examines the change in ratings from the 
first observed rating in the analysis period to the last observed rating in the analysis period. Rating level changes are reported for 
the full period of 2012 to 2016 for seven states, and 2014 to 2016 for California and Ohio. California data are from 16 counties 
that implemented TQRIS beginning in 2012; all other data are statewide. Rating categories reflect the order of ratings within states’ 
rating structure, and treat states with four levels (Massachusetts and Minnesota) as having no middle level. 


Figure reads: Four percent of state-funded pre-K programs were first observed at the lowest rating level and remained at that rating for the 
entire study period. 


pre-K = prekindergarten; RTT-ELC = Race to the Top-Early Learning Challenge; TQRIS = tiered quality rating and improvement systems. 
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For more information on the full study, please visit: 


https://ies.ed.gov/ncee/projects/evaluation/other racetotop.as 
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